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Abstract 


The  North  Atlantic  Treaty  Organization  (NATO)  Land  Capability  Group  1  (LCG  1)  has  identified 
a  need  to  develop  and  validate  symbology  for  the  individual  soldier  that  can  be  displayed  on  a 
wide  range  of  digital  displays.  Prototype  symbology  sets  for  this  purpose  have  been  developed 
by  several  NATO  nations.  A  heuristic  evaluation  of  these  sets  indicated  that  none  of  them  were 
entirely  consistent  with  good  human  factors  practice.  A  new  set  (CHARLIE)  was  developed  that 
conformed  more  closely  to  human  factors  guidelines.  To  further  evaluate  the  most  suitable 
symbology,  an  empirical  assessment  was  carried  out.  Two  different  sizes  of  three  prototype 
symbol  sets  were  evaluated  against  backgrounds  composed  of  digital  photographic  images  of 
differing  complexity  and  digitized  paper  maps  under  two  different  display  resolutions.  The 
symbol  sets  evaluated  were  the  CHARLIE  set  discussed  above,  and  two  variants  (ALPHA  and 
BRAVO)  of  a  set  designed  by  NATO  LCG1.  The  symbols  were  evaluated  using  a  visual  search 
task  with  accuracy  and  speed  of  response  as  the  measures  of  performance.  The  results  indicated 
that  detection  and  discrimination  of  the  CHARLIE  symbols  was  significantly  faster  under  all  of 
the  conditions  tested  and  significantly  more  accurate  with  low  resolution  displays.  Moreover, 
performance  with  the  CHARLIE  symbols  was  consistent  across  all  the  conditions  tested 
(background,  resolution,  and  symbol  size). 


Resume 


Le  lerGroupe  sur  les  capacites  terrestres  (LCG  1)  de  l’Organisation  du  Traite  de  l’Atlantique 
Nord  (OTAN)  a  ceme  la  necessity  de  mettre  au  point  et  de  valider  une  symbologie  du  soldat  qui 
peut  etre  affichee  sur  une  vaste  gamme  d’ecrans  numeriques.  Plusieurs  pays  de  l’OTAN  ont 
developpe  des  prototypes  d’ensembles  de  symboles  a  cette  fin.  Une  evaluation  heuristique  de  ces 
ensembles  a  indique  qu’aucun  d’eux  n’etait  tout  a  fait  conforme  aux  bonnes  pratiques 
ergonomiques.  On  a  elabore  un  nouvel  ensemble  (CHARLIE)  qui  respectait  plus  etroitement  les 
lignes  directrices  de  l’ergonomie.  Pour  evaluer  davantage  la  symbologie  la  plus  appropriee,  on  a 
precede  a  une  evaluation  empirique.  Deux  tailles  differentes  de  trois  prototypes  d’ensembles  de 
symboles  ont  ete  evaluees  sur  des  fonds  composes  d’images  photographiques  numeriques  de 
complexity  differente  et  de  cartes  papier  numerisees,  selon  deux  resolutions  d’affichage 
differentes.  L’evaluation  a  porte  sur  l’ensemble  CHARLIE  susmentionne  et  sur  deux  variantes 
(ALPHA  et  BRAVO)  d’un  ensemble  congu  par  le  LCG  1  de  l’OTAN.  On  a  evalue  les  symboles 
au  moyen  d’une  recherche  visuelle  en  utilisant  comme  mesures  de  rendement  la  precision  et  la 
vitesse  de  reponse.  Les  resultats  ont  indique  que  la  detection  et  la  discrimination  des  symboles 
CHARLIE  etaient  beaucoup  plus  rapides  dans  toutes  les  conditions  d’essai  et  beaucoup  plus 
precises  avec  des  affichages  a  basse  resolution.  Par  ailleurs,  les  symboles  CHARLIE  ont  donne 
lieu  a  un  rendement  uniforme  dans  toutes  les  conditions  d’essai  (fond,  resolution  et  taille  des 
symboles). 
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Executive  summary 


Development  and  evaluation  of  symbology  to  identify  individual 
dismounted  soldiers: 

Sharon  M.  McFadden;  Jeremy  Robson;  David  Tack;  DRDC  Toronto  CR  2010- 
152;  Defence  R&D  Canada  -  Toronto;  December  2011. 

Introduction  or  background:  The  North  Atlantic  Treaty  Organization  (NATO)  Land  Capability 
Group  1  (LCG  1)  has  identified  a  need  to  develop  and  validate  symbology  for  the  individual 
soldier  that  can  be  displayed  on  a  wide  range  of  digital  displays.  Prototype  symbology  sets  for 
this  purpose  have  been  developed  by  several  NATO  nations.  A  heuristic  evaluation  of  these  sets 
indicated  that  none  of  them  were  entirely  consistent  with  good  human  factors  practice.  A  new 
design  concept  (CHARLIE)  was  proposed  that  conformed  more  closely  to  human  factors 
guidelines. 

To  further  evaluate  the  most  suitable  symbology,  an  empirical  assessment  was  carried  out.  The 
goal  was  to  assess  the  detectability  of  two  different  versions  (Small  and  Large  symbols)  of  three 
prototype  symbol  sets  against  varied  backgrounds,  composed  of  digital  photographic  images  of 
differing  complexity  and  digitized  paper  maps,  presented  at  two  different  display  resolutions 
(high  and  low).  The  symbol  sets  evaluated  were  the  CHARLIE  set  discussed  above,  and  two 
variants  (ALPHA  and  BRAVO)  of  a  set  designed  by  NATO  LCG1.  With  ALPHA,  affiliation  and 
echelon  were  presented  against  a  white  background  and  with  BRAVO,  they  were  presented 
against  the  display  background.  The  symbols  were  evaluated  using  a  visual  search  task  with 
accuracy  and  speed  of  response  as  the  measures  of  performance. 

Results:  The  results  indicated  that  the  CHARLIE  symbols  were  detected  and  discriminated 
significantly  faster  under  all  of  the  conditions  tested  and  significantly  more  accurately  with  low 
resolution  displays.  Moreover,  performance  with  the  CHARLIE  symbols  was  consistent  across  all 
the  conditions  tested  (background,  resolution,  and  symbol  size).  However,  response  times  did 
vary  across  the  different  symbols  within  the  CHARLIE  set. 

Significance:  The  results  of  this  study  indicate  that  it  is  possible  to  design  symbols  that  are 
legible  under  different  display  resolutions  and  visible  against  a  wide  range  of  backgrounds. 
However,  the  variability  in  performance  across  the  different  symbols  suggests  that  discrimination 
could  deteriorate  as  the  number  of  different  symbols  on  the  screen  increases. 

Future  plans:  The  first  step  is  to  extend  the  CHARLIE  symbol  set  to  represent  all  NATO 
soldiers.  Once  that  is  done,  it  will  be  necessary  to  re-evaluate  the  symbols  to  determine  the  effect 
of  increasing  the  number  of  different  symbols  and  methods  for  optimizing  discriminability.  In 
addition,  it  will  be  important  to  evaluate  the  utility  of  the  proposed  symbols. 
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dismounted  soldiers: 

Sharon  M.  McFadden;  Jeremy  Robson;  David  Tack;  DRDC  Toronto  CR  2010- 
152;  R&D  pour  la  defense  Canada  -  Toronto;  December  2011. 

Introduction  ou  contexte  :  Le  lerGroupe  sur  les  capacites  terrestres  (LCG  1)  de  reorganisation 
du  Traite  de  l’Atlantique  Nord  (OTAN)  a  ceme  la  necessite  de  mettre  au  point  et  de  valider  une 
symbologie  du  soldat  qui  peut  etre  affichee  sur  une  vaste  gamme  d’ecrans  numeriques.  Plusieurs 
pays  de  l’OTAN  ont  developpe  des  prototypes  d’ensembles  de  symboles  a  cette  fin.  Une 
evaluation  heuristique  de  ces  ensembles  a  indique  qu’aucun  d’eux  n’etait  tout  a  fait  conforme  aux 
bonnes  pratiques  ergonomiques.  On  a  propose  un  nouveau  concept  (CHARLIE)  qui  respectait 
plus  etroitement  les  lignes  directrices  de  l’ergonomie. 

Pour  evaluer  davantage  la  symbologie  la  plus  appropriee,  on  a  precede  a  une  evaluation 
empirique.  Le  but  consistait  a  evaluer  la  detectabilite  de  deux  versions  differentes  (petits  et  grands 
symboles)  de  trois  prototypes  d’ensembles  de  symboles  sur  des  fonds  varies,  composes  d’images 
photographiques  numeriques  de  complexity  differente  et  de  cartes  papier  numerisees,  selon  deux 
resolutions  d’affichage  differentes  (haute  et  basse).  L’ evaluation  a  porte  sur  1’ ensemble 
CHARLIE  susmentionne  et  sur  deux  variantes  (ALPHA  et  BRAVO)  d’un  ensemble  conqu  par  le 
LCG  1  de  l’OTAN.  ALPHA  presentait  l’affiliation  et  l’echelon  sur  un  fond  blanc,  tandis  que 
BRAVO  les  presentait  sur  le  fond  d’ecran.  On  a  evalue  les  symboles  au  moyen  d’une  recherche 
visuelle  en  utilisant  comme  mesures  de  rendement  la  precision  et  la  vitesse  de  reponse. 

Resultats  :  Les  resultats  ont  indique  que  la  detection  et  la  discrimination  des  symboles 
CHARLIE  etaient  beaucoup  plus  rapides  dans  toutes  les  conditions  d’essai  et  beaucoup  plus 
precises  avec  des  affichages  a  basse  resolution.  Par  ailleurs,  les  symboles  CHARLIE  ont  donne 
lieu  a  un  rendement  uniforme  dans  toutes  les  conditions  d’essai  (fond,  resolution  et  taille  des 
symboles).  Les  temps  de  reponse  variaient  toutefois  entre  les  differents  symboles  dans  1’ ensemble 
CHARLIE. 

Importance  :  Les  resultats  de  cette  etude  indiquent  qu’il  est  possible  de  concevoir  des  symboles 
lisibles  selon  differentes  resolutions  d’affichage  et  visibles  sur  un  large  eventail  de  fonds. 
Cependant,  la  variability  du  rendement  entre  les  differents  symboles  laisse  supposer  que  la 
discrimination  pourrait  se  deteriorer  a  mesurer  qu’augmente  le  nombre  de  symboles  differents  sur 
l’ecran. 

Projets  futurs  :  La  premiere  etape  consiste  a  elargir  l’ensemble  de  symboles  CHARLIE  de 
maniere  a  representer  tous  les  soldats  de  l’OTAN.  Ensuite,  il  faudra  reevaluer  les  symboles  afm 
de  determiner  l’effet  de  l’augmentation  du  nombre  de  methodes  et  symboles  differents  pour 
optimiser  la  discriminabilite.  En  outre,  il  sera  important  d’evaluer  l’utilite  des  symboles  proposes. 
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Introduction 


Background 

Future  soldier  systems  are  being  developed  that  can  display  a  soldier's  geo-location  on  a  digital 
map  or  digital  photograph,  and  present  the  resulting  image  through  a  helmet-mounted  display, 
weapon  sight  display,  portable  digital  assistant,  mini-tablet,  or  laptop.  Many  nations  intend  to 
provide  blue -force  tracking  on  their  hand-held  soldier  system  computers  so  that  the  positions  of 
all,  or  select  members  of  a  unit,  can  be  observed  in  real  time  on  the  same  digital  map  display. 

A  North  Atlantic  Treaty  Organization  (NATO)  standard  for  military  symbols  for  land  systems 
(APP-6A)  (North  Atlantic  Treaty  Organization  1999)  exists  to  provide  common  NATO 
operational  symbology  for  interoperability  of  Land  Command,  Control,  Communications, 
Computer,  and  Intelligence  (C4I)  systems.  The  conventions  in  NATO  APP-6A  are  derived  from 
M1L-STD-2525C  (Department  of  Defense  2008).  This  standard  provides  a  common  symbol 
hierarchy,  information  taxonomy,  and  symbol  identifiers,  including  a  standard  symbol  set  for  all 
future  NATO  C4I  operations  in  the  force  and  engagement  domains.  Unfortunately,  neither  APP- 
6A  nor  M1L-STD-2525C  includes  symbology  to  identify  individual  soldiers.  Consequently, 
NATO  Land  Capabilities  Group  1  (LCG  1)  has  identified  a  need  to  develop  and  validate  a 
standard  NATO  symbol  set  that  cover  all  NATO  dismounted  soldier  roles. 

Several  NATO  nations  have  developed  soldier-level  symbology  for  their  own  purposes.  Example 
symbols,  from  the  sets  that  were  provided  to  LCG  1  for  evaluation,  are  shown  in  Table  1.  In 
addition,  LCG  1  also  put  forward  a  design  concept  that  was  similar  to,  but  more  extensive  than, 
set  6.  Examples  of  all  the  available  symbols  are  shown  in  Annex  A.  None  of  the  proposed  symbol 
sets  had  been  systematically  evaluated.  At  the  request  of  LCG  1,  Defence  Research  and 
Development  Canada  (DRDC)  -  Toronto  undertook  a  project  to  systematically  assess  these 
existing  sets  in  order  to  validate  the  selection  of  either  an  existing  set,  a  modified  version  of  an 
existing  set,  or  a  new  set  that  better  complies  with  human  factors  guidelines. 
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Table  1:  Examples  of  symbols  proposed  by  different  NATO  nations 


Symbol  set 

Sample  symbol 

Number  of  available 
symbols 

1 

J 

2 

2 

5 

8 

3 

1 

3 

4 

6 

4 

5 

S 

mith 

ft 

2 

6 

6 

7 

Evaluation  framework 

As  a  first  step,  a  framework  was  developed  to  evaluate  the  existing  symbol  sets.  To  avoid  bias, 
the  framework  was  based  on  international  standards1.  It  was  designed  to  meet  the  following 
criteria: 

•  Military  domain  -  any  candidate  symbol  set  must  uphold  the  design  conventions 
(particularly  Land  Force)  that  have  long  been  in  practice  in  NATO  nations  and  be  effective 
in  the  military  domain  in  which  they  will  be  used. 

•  Human  factors  domain  -  any  candidate  symbol  set  must  meet  human  factors  criteria  for 
symbol  design  to  ensure  optimum  detectability,  legibility,  and  usability. 

•  Adaptability  to  digital  displays  -  candidate  symbols  should  be  usable  on  a  wide  range  of 
display  technology  with  different  physical  parameters. 


1  For  a  list  of  the  standards  reviewed,  see  Tack,  Robson  and  Matthews  (2009). 
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Together,  these  three  domain  areas  were  believed  to  encompass  the  broad  range  of  symbol  set 
usability  and  employment  issues  for  small  unit  army  applications. 


Military  domain 

APP-6A  sets  out  a  number  of  symbol  design  conventions  that  need  to  be  maintained  in  any  future 
soldier  symbology.  These  include  the  frame  shape,  affiliation,  status,  fill,  interior  icons,  and 
modifiers  for  movement,  echelon,  equipment,  and  so  on.  Specific  guidance  is  provided  for 
symbol  framing,  placement  of  icons  and  modifiers  and  design  specifications  are  provided  for  size, 
shape,  orientation,  colour,  line  width,  and  so  on.  It  is  important  that  any  symbols  representing 
individual  soldiers  be  consistent  with  these  existing  criteria  to  simplify  learning  and  avoid 
confusion. 

In  addition,  it  was  determined  that  to  provide  a  comprehensive  set  of  symbols  for  soldier 
identification  at  Company  level  and  below,  it  was  important  to  include  symbols  that  incorporated 
the  following  dimensions: 

•  Role:  military  function  of  the  individual,  e.g.,  commander,  fire  team  leader. 

•  Echelon:  Military  operational  unit,  e.g.,  company,  platoon,  section,  squad. 

•  Affiliation:  The  unit  and  sub-unit  that  employs  the  soldier. 

Using  all  three  dimensions  enables  a  symbol  to  specifically  identify  almost  any  soldier  on  the 
battlefield,  assuming  an  intra-section  naming  convention  for  soldiers  within  each  sub-unit. 

Finally,  the  context  in  which  the  symbols  will  be  presented  was  considered.  Operations  are 
increasingly  occurring  in  built  up  areas  with  complex  terrain.  This  introduces  more  complexity 
into  the  digital  maps  and  satellite  photographs  that  will  likely  form  the  backgrounds  of  map 
displays  in  future  soldier  systems.  As  well,  it  will  be  possible  to  superimpose  a  wide  range  of  data 
on  these  backgrounds.  This  will  lead  to  maps  being  cluttered  with  many  different  symbols  for 
friendly  and  enemy  forces,  equipment  and  structures.  It  is  important  that  the  visibility  of  the 
symbols  not  be  reduced  by  background  clutter  or  because  of  similarity  to  other  critical 
information. 


Human  factors  domain 

A  wide  range  of  standards  have  been  developed  by  industry  and  government  to  support  the  design 
of  symbols  and  icons.  Based  on  a  review  of  these,  especially  EG  201  379  VI.  1.1  (European 
Telecommunication  Standards  Institute  1998),  the  following  human  factors  criteria  were 
determined  to  be  important  in  evaluating  the  utility  and  usability  of  symbols: 

•  Comprehensibility  -  the  ease  with  which  the  meaning  of  the  symbol  is  understood. 

•  Discriminability  -the  ease  with  which  a  given  symbol  can  be  distinguished  from  other 
symbols  that  might  occur  in  close  spatial,  temporal  or  contextual  proximity. 

•  Leamability  -  the  ease  with  which  the  meaning  of  a  symbol  can  be  recalled  after  it  has  been 
understood. 
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•  Legibility  -  the  ease  with  which  the  figural  detail  of  a  symbol  can  be  discerned. 

•  Recognisability  -  the  ease  with  which  it  is  possible  to  identify  a  symbol  based  on  previous 
experiences  with  the  same  or  similar  types  of  symbols. 


Adaptability  to  digital  displays 

It  is  likely  that  the  proposed  soldier  symbology  will  be  displayed  on  a  wide  range  of  displays  with 
different  characteristics  including  screen  resolution,  screen  size,  and  technology.  ISO  80416-4 
(International  Standards  Organization  2005)  was  used  to  derive  digital  display  evaluation  criteria 
and  provided  guidelines  for  the  adaptation  of  symbols  to  screens  and  displays  for  a  wide  range  of 
equipment  and  devices.  Specifically,  issues  associated  with  symbol  size,  pixel  grids,  degradation 
of  resolution,  and  use  of  colour  were  considered. 


Evaluations 

With  the  possible  exception  of  the  LCG1  set,  none  of  the  symbol  sets  included  symbols  for  all  of 
the  NATO  roles  and  echelons.  This  made  it  difficult  to  conduct  a  systematic  empirical  evaluation. 
In  addition,  it  would  have  been  time  consuming  to  run  an  empirical  study  that  compared  all  of  the 
symbols  against  the  framework.  Thus  the  evaluation  was  divided  into  two  parts.  Part  1  involved  a 
heuristic  evaluation  of  each  of  the  available  symbols  sets.  As  a  result  of  this  evaluation,  a  new 
design  concept  was  proposed  that  more  closely  conformed  to  the  criteria  used  in  the  heuristic 
evaluation.  Part  2  involved  an  empirical  evaluation  of  the  new  symbol  set  against  the  symbol  set 
proposed  by  LCG1.  It  was  chosen  because  it  included  the  widest  range  of  symbols. 
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Part  1:  Heuristic  evaluation 


Method 

For  the  heuristic  evaluation,  specific  criteria  were  extracted  from  the  standards  and  guidelines 
used  in  developing  the  heuristic  framework  under  the  three  domains  discussed  previously.  Each 
symbol  set  was  then  evaluated  by  three  human  factors  experts  against  each  of  these  criteria  using 
a  five  point  scale.  The  specific  criteria  assessed  under  each  of  the  three  domains  in  the  evaluation 
framework  were  as  follows: 

•  Military  criteria 

1.  All  symbols  within  the  set  conform  to  a  rectangular  “ground”  shape  as  per  the  aspect 
ratios  of  M1L-STD-2525C  (Department  of  Defense  2008)  (i.e.  ratios  of  1:1,  1:1.5). 

2.  Symbols  have  an  appropriate  frame  border  width. 

3.  Symbols  use  both  shape  and  fill  colour  to  indicate  battle  dimension. 

4.  Symbol  friendly  interior  elements  can  be  recognizable  at  the  smallest  expected  symbol 
size. 

5.  Symbol  interior  elements  can  be  recognizable  at  the  smallest  expected  symbol  size  for  the 
following  affiliations:  red  (hostile  and  suspect),  yellow  (unknown),  cyan  (friendly)  and 
green  (neutral). 

6.  The  frame  indicates  the  location  status  (actual  versus  planned)  of  the  soldier  at  the 
smallest  expected  symbol  size. 

7.  Symbols  are  universally  consistent  with  NATO  echelon  conventions. 

•  Human  factors  criteria 

1 .  Role  symbols  are  discriminable  from  each  other. 

2.  Symbols  maintain  their  integrity  when  clustered  together  (not  overlapped). 

3.  Symbols  convey  required  information  with  fewest  elements. 

4.  Symbols  are  replicated  easily  (i.e.,  drawn  by  hand  on  paper). 

5.  Symbols  are  readily  discemable  from  a  temperate  map  background. 

6.  Symbols  are  readily  discemable  from  an  arid  map  background. 

7.  Symbols  are  readily  discemable  from  highly  textured  map  background  (lines, 
buildings,  etc). 

8.  For  discrete  icons  there  is  a  clear  gap  between  the  icon  and  the  surrounding  border  at 
the  smallest  usable  size. 
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9.  Symbol  can  accommodate  worst-case  overlapping  direction  without  loss  of 
discemability. 

•  Digital  display  criteria 

1.  The  role  of  symbols  can  be  discriminated  from  each  other  at  smaller  sizes  and  at 
lower  resolutions. 

2.  The  echelon  of  symbols  can  be  discriminated  from  each  other  at  smaller  sizes  and  at 
lower  resolutions. 

3.  The  unit  affiliation  of  symbols  can  be  discriminated  from  each  other  at  smaller  sizes 
and  at  lower  resolutions. 

To  assess  criteria  5  to  7  in  the  human  factors  domain,  representative  symbols  were  superimposed 
on  digital  imagery  of  varying  complexity  and  digitized  maps.  To  assess  the  symbol  sets  against 
the  digital  display  criteria,  a  complex  example  symbol  from  each  set  was  adjusted  in  size  and 
resolution  (Table  2  provides  an  example).  For  size,  the  symbols  were  adjusted  to  fit  into  squares 
of  the  following  dimensions: 

•  20mm  x  20mm 

•  1 5mm  x  15mm 

•  10mm  x  10mm 

•  5mm  x  5mm 

For  resolution,  symbols  were  graphically  re-sampled  into  the  following  four  pixel  grids,  as 
suggested  by  ISO  80416-4  (International  Standards  Organization  2005): 

•  64  x  64  pixels 

•  32  x  32  pixels 

•  16x16  pixels 

•  12x12  pixels 
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Table  2:  Sample  symbol  at  each  combination  of  size  and  resolution  used  in  the  evaluation 
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Results 

The  results  for  the  three  sets  of  criteria,  military,  human  factors,  and  display,  are  shown  in  Figure 
1.  Given  the  quality  and  quantity  of  data,  any  type  of  statistical  analysis  would  be  inappropriate. 
For  the  first  two  factors,  scores  for  each  set  were  summed  to  give  a  general  measure  of  goodness- 
of-fit.  For  the  display  domain,  individual  scores  are  shown.  Neither  set  1  nor  set  4  contained  any 
interior  elements.  As  a  result,  they  could  not  be  evaluated  on  criteria  4  and  5  under  the  military 
domain  nor  criterion  8  under  the  human  factors  domain.  To  assess  the  impact  of  this  difference, 
scores  with  and  without  these  criteria  were  computed  for  the  remaining  candidates.  In  general, 
removing  these  questions  tended  to  reduce  the  average  scores  on  the  military  criteria,  but  had  no 
effect  on  the  average  scores  for  the  human  factors  criteria.  Thus  for  the  military  criteria  total  score 
with  and  without  these  questions  have  been  computed,  and  for  the  human  factors  criteria, 
criterion  8  has  not  been  included  in  the  total  score. 
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Figure  1:  Scores  for  the  existing  symbol  sets  on  the  military,  human  factors,  and  display  criteria. 
The  dotted  line  indicates  a  perfect  score  for  five  military  criteria. 
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The  display  criteria  focused  on  the  discriminability  of  role,  echelon,  and  affiliation  information. 
Unfortunately  one  symbol  set  did  not  provide  role  information;  two  did  not  provide  echelon 
information,  and  only  one  included  affiliation  information. 

As  can  be  seen,  symbol  set  3  tended  to  perform  best.  In  addition  to  locating  all  components  inside 
the  framework,  it  complied  most  closely  with  APP-6A  /  MIL  STD  2525C.  Unfortunately,  it  does 
not  address  the  challenge  of  covering  the  required  range  of  echelons  and  affiliations.  Symbol  set  5 
tended  to  perform  the  poorest  across  all  three  areas.  Unlike  the  other  sets,  it  employs  a  complex 
torso  shape  instead  of  a  simple  geometric  shape.  The  torso  shape  leaves  less  room  for  echelon  and 
affiliation  information.  On  the  other  hand,  it  was  the  only  symbol  set  that  included  role,  echelon, 
and  affiliation  information.  The  primary  weakness  with  all  but  set  3  was  the  use  of  text  and  or 
lines  outside  the  framework.  Text  and  lines  may  become  confused  with  background  features 
especially  in  urban  maps  and  satellite  imagery. 

Proposed  symbol  set 

As  a  result  of  the  problems,  discussed  in  the  previous  paragraph,  with  the  existing  symbol  sets,  it 
was  decided  to  develop  a  concept  for  a  symbol  set  that  would  more  closely  conform  to  the 
heuristic  framework.  Thus,  as  shown  in  Table  3,  the  symbols  met  the  following  design  criteria: 

•  Symbol  shape  and  in-fill:  Symbols  have  a  rectangular  shape  with  a  clear  frame  border  and 
light  blue  in-fill  as  per  APP-6A. 

•  Role:  Officers  are  denoted  by  a  regular  frame  while  enlisted  soldiers  are  denoted  by  the 
addition  of  angled  comers.  In  sourcing  a  way  to  denote  enlisted  positions,  the  rank  insignia 
for  a  range  of  NATO  nations  was  consulted.  The  characteristic  of  most  enlisted  rank 
insignia  is  a  chevron  or  angled  tabs.  The  angled  comers  of  the  enlisted  symbols  give  the 
illusion  of  a  chevron  over  the  blue  in-fill. 

•  Echelon:  Echelon  is  represented  using  the  same  convention  as  APP-6A,  as  indicated  in 
Table  4  below. 

•  Affiliation:  Unit  affiliation  is  indicated  by  the  convention  of  "unit  of  sub-unit".  For 
example,  1-3  denotes  1  Platoon,  3  Section.  Soldier  members  of  a  Section/Squad,  not 
including  the  Section  Commander  or  Second  in  Command  (2i/c),  would  be  denoted  by  a 
single  number  to  reflect  their  position. 

•  Clutter  and  Overlap:  All  soldier  identity  information  is  contained  within  the  symbol  frame 
to  avoid  any  loss  of  discemability  due  to  background  clutter  and  to  minimize  any 
information  loss  due  to  symbol  overlapping. 

The  new  symbol  set  concept  design  was  scored  using  the  analytical  framework  and  generated  a 
near-perfect  score.  The  design  only  lost  points  for  discemability  of  affiliation  at  the  lowest 
resolutions  and  sizes,  suggesting  that  some  fine  tuning  of  numeric  text  fonts  is  required. 
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Table  3:  Proposed  concept  for  a  symbol  set 


NATO  position 

Symbol  concept 

Company  Commander 

1 

2  -  7 

Company  Second-in-Command 

r 

3-1 

Fire  Team  Leader 

<2 

1  -  1 

Platoon  Commander 

•  •  • 

2  -  7 

Platoon  Sergeant 

1  -  2 

Section  Commander 

•  • 

3  -  2 

Section  Second-in-Command 

3-2 

Soldier 

3 

1  -3 

Table  4:  NATO  APP-6A  echelon  icons 


Indicator 

Description 

0 

Team/Crew 

• 

Squad 

•  • 

Section 

•  •• 

Platoon 

Company 
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Part  2:  Empirical  evaluation 


Since  the  new  design  concept  was  based  on  the  heuristic  framework,  it  was  not  surprising  that  it 
scored  better  than  the  remaining  design  concepts  in  the  heuristic  evaluation.  The  next  step  was  to 
conduct  a  more  objective  evaluation.  One  of  the  limitations  of  the  previous  comparison  was  that 
the  symbol  sets  varied  in  the  range  of  roles  and  echelons  they  included.  Ideally,  we  wanted  to 
evaluate  as  rich  a  symbol  set  as  possible  in  order  to  assess  the  discriminability  of  the  individual 
symbols  within  a  set.  Of  the  symbol  sets  originally  evaluated,  only  symbol  set  7,  developed  by 
LCG1,  covered  the  same  range  of  roles  and  echelons  as  the  proposed  symbol  set.  However,  even 
it  did  not  include  a  concept  for  affiliation.  To  overcome  this  limitation,  a  design  concept 
consistent  with  APP-6A  was  used;  namely,  affiliation  was  specified  using  text  in  the  upper  left 
comer  of  the  symbol  outside  its  frame.  Two  different  variants  were  evaluated:  ALPHA  which 
placed  the  affiliation  and  echelon  information  against  a  white  background  in  accordance  with 
NATO’s  design  guidelines  (North  Atlantic  Treaty  Organization  1999)  and  BRAVO  which 
followed  M1L-STD-2525C.  The  proposed  symbol  set  was  labelled  CHARLIE. 

Table  5:  Examples  of  the  symbols  used  in  the  evaluation. 
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The  actual  symbols  used  in  the  experiment  are  shown  in  Table  5.  The  design  concept  for  the 
CHARLIE  symbols  has  been  described  earlier.  For  BRAVO  and  ALPHA,  role  is  indicated  by  the 
presence  (enlisted  role)  or  absence  (officer  role)  of  one  or  more  diagonal  lines  inside  the  blue 
circle.  Echelon  is  denoted  by  lines  and  cross  lines  extending  above  the  circle  and  affiliation  by  the 
convention  of  unit  of  sub-unit  (e.g.  13  indicates  the  soldier  is  a  member  of  One  Platoon,  Three 
Section)  beside  the  echelon  component2. 


2 

It  should  be  noted  that  the  symbol  sets  under  evaluation  are  prototypes  and  the  design  concepts  used  in 
their  construction  may  not  be  applied  consistently  across  all  symbols.  Further  development,  aided  by  the 
results  of  this  study,  is  required  to  produce  symbols  that  cover  the  roles,  echelons  and  affiliations  of  all 
NATO  soldiers. 
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To  assess  the  discriminability  and  recognisability  of  the  three  design  concepts,  performance  was 
measured  using  two  variants  of  a  visual  search  task.  In  both  tasks,  observers  were  required  to 
search  for  a  specified  symbol  in  a  background  of  distractor  symbols.  In  the  first  task,  the 
distractor  symbols  always  differed  in  either  role  or  echelon.  Thus  if  the  target  was  Platoon 
Commander  1-3,  none  of  the  distractor  symbols  represented  a  Platoon  Commander  from  a 
different  unit  or  sub  unit.  In  the  second  task,  some  of  the  distractors  differed  from  the  target 
symbol  only  in  unit  and/or  sub  unit. 

While  both  tasks  assessed  discriminability  and  recognisability,  the  latter  task  also  assessed 
legibility.  To  further  assess  legibility,  the  study  examined  performance  at  two  different  symbol 
sizes  and  two  different  display  resolutions.  The  smaller  symbol  size  was  based  on  the  outcome  of 
the  heuristic  evaluation  (Tack  et  al.  2009).  The  larger  symbol  size  was  included  to  determine  the 
advantage/disadvantage  of  increasing  symbol  size  even  further.  While  a  larger  symbol  may  be 
more  visible  against  a  complex  background,  it  can  also  result  in  more  frequent  symbol  overlap. 
To  minimize  symbol  overlap,  the  increase  was  limited  to  20%.  Since  these  proposed  symbols  are 
likely  to  be  displayed  on  a  wide  range  of  technologies,  it  is  important  that  the  symbols  remain 
legible  at  different  display  resolutions.  Thus,  performance  was  assessed  at  two  display 
resolutions.  The  high  resolution  condition  (72  pixels  per  inch)  was  equivalent  to  a  standard,  1024 
by  768,  monitor.  The  lower  resolution  (50  pixels  per  inch)  approximated  a  smaller,  lower 
resolution,  640  by  480,  monitor. 

The  final  variable  assessed  in  the  current  study  was  background.  The  backgrounds  used  in  future 
soldier  systems  are  likely  to  vary  from  very  simple,  e.g.,  desert  and  rural  areas  through  to 
complex  urban  environments  with  high  density  of  buildings,  roads,  and  other  structures  as  well  as 
digitized  conventional  maps.  Moreover,  the  results  of  the  heuristic  evaluation  indicated  that  the 
visibility  of  the  external  modifiers  used  to  define  echelon  with  ALPHA  and  BRAVO  was  reduced 
when  the  symbols  were  presented  against  a  complex  background.  For  that  reason,  the  symbol 
sets  were  evaluated  against  three  types  of  backgrounds:  Simple,  Complex,  and  Map.  Simple 
backgrounds  were  digital  photographs  that  had  little  variation  in  colour  or  texture,  and  contained 
few  buildings,  roads  etc.  Complex  backgrounds  were  also  digital  photographs,  but  they  had  more 
variability  and  irregularity  through  increased  colour  differentiation,  less  regularity  in  terms  of 
physical  features  (roads,  buildings,  topography  etc),  greater  contrast,  and  more  buildings.  Map 
backgrounds  consisted  of  digitally  scanned  military  paper  maps. 


Method 

Participants 

Twenty-five  volunteers,  1 1  males  and  14  females,  participated  in  the  experiment.  They  ranged  in 
age  from  1 8  to  60  with  a  mean  age  of  25.5.  Participants  were  required  to  self  identify  that  they 
had  normal  colour  vision  and  normal  or  corrected  to  normal  visual  acuity. 


Experimental  Design 

For  the  first  (detection)  task,  a  3  (symbol  set)  by  3  (background  type)  by  2  (symbol  size)  by  2 
(display  resolution)  factorial  design  was  used.  Each  of  the  seven  target  symbols  within  each  set 
was  presented  against  three  different  types  of  background  -  Simple,  Complex  and  Map.  Two 
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symbol  sizes  were  used  for  each  symbol  set:  Small  symbols  (defined  using  a  heuristic  (Tack  et  al. 
2009))  and  a  second  set  of  symbols  20%  larger  than  the  Small  symbols  and  referred  to  throughout 
this  report  as  Large.  Finally,  the  backgrounds  and  symbols  were  presented  at  two  different 
resolutions,  HR  (72  pixels  per  inch)  and  LR  (50  pixels  per  inch). 

Participants  completed  a  run  of  7  trials  under  each  combination  of  symbol  set,  resolution,  symbol 
size,  and  background  for  a  total  of  252  trials.  On  each  trial  in  a  run,  one  of  the  seven  symbol 
shown  in  Table  5  (e.g.,  Platoon  Sergeant  (1-3)  or  Fire  Team  Leader  (2-2)),  for  the  symbol  set 
under  study  in  that  session,  was  selected  as  the  target.  It  was  presented  together  with  1 9  distractor 
symbols.  The  distractors  were  randomly  selected  from  the  remaining  symbols  in  that  set  with  the 
restriction  that  they  could  not  have  the  same  role  and  echelon  as  the  target  symbol  (See  Annex  B 
for  examples  of  all  possible  symbols  that  might  appear  as  distractor  symbols).  The  target  symbols 
on  each  of  the  7  trials  represented  a  different  symbol  role  and/or  echelon  and  the  order  of 
presentation  of  the  different  target  symbols  was  randomly  varied  across  the  different  conditions 
and  participants. 

For  the  second  (legibility)  task  a  3  (symbol  set)  by  3  (background  type)  by  2  (symbol  size) 
factorial  design  was  used.  Due  to  limitations  in  time  and  resources,  it  was  not  carried  out  in  the 
low  resolution  condition.  Otherwise,  the  legibility  task  was  identical  to  the  detection  task  except 
that  symbols  representing  the  same  role  and  echelon  as  the  target  symbol,  but  with  different 
affiliation  information,  were  included  among  the  distractor  symbols.  For  example,  if  the  target 
symbol  was  Fire  Team  Leader  1-1,  the  display  would  also  feature  the  symbols  for  Fire  Team 
Leader  2-2.  The  purpose  of  this  second  run  was  to  measure  the  legibility  of  the  modifier 
information  on  the  symbol  (e.g.,  1-1).  As  will  be  noted  in  Table  5,  the  instantiation  of  this 
information  varied  considerably  across  the  three  symbol  sets. 

Presentation  order  of  the  three  symbol  sets  was  randomized  across  participants  and  each  symbol 
set  was  presented  on  a  different  day.  The  order  of  the  remaining  conditions  was  fixed  (see  Table 
6).  Given  the  relatively  small  number  of  trials,  it  was  decided  that  it  was  preferable  to  present  first 
those  conditions  in  which  the  symbols  were  likely  to  be  the  most  discriminable. 
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Table  6:  Presentation  order  of  the  different  conditions  for  each  symbol  set. 


Resolution 

Symbol  size 

Background 

Number  of  trials 

HR 

Large 

Simple 

7  detection 

7  legibility 

Complex 

7  detection 

7  legibility 

Paper  map 

7  detection 

7  legibility 

Small 

Simple 

7  detection 

7  legibility 

Complex 

7  detection 

7  legibility 

Paper  map 

7  detection 

7  legibility 

LR 

Large 

Simple 

7  detection 

Complex 

7  detection 

Paper  map 

7  detection 

Small 

Simple 

7  detection 

Complex 

7  detection 

Paper  map 

7  detection 

Apparatus 

The  apparatus  comprised  a  standard  “Windows”  workstation  with  a  19”  colour  screen,  a  mouse 
and  keyboard.  Participants  were  also  provided  with  a  work  surface  to  record  notes,  and  an 
ergonomically  designed  operator’s  chair. 

Stimuli 

The  stimuli  were  the  symbols  in  the  three  symbol  sets,  ALPHA,  BRAVO  and  CHARLIE,  shown 
in  Annex  B.  Each  symbol  shows  the  role,  echelon,  and  affiliation  of  a  dismounted  soldier.  The 
Small  ALPHA  symbols  were  8mm  wide  by  7mm  high  and  the  Large  were  9  by  8mm.  Small 
BRAVO  symbols  were  7mm  wide  by  7mm  high  and  the  Large  were  8  by  8mm.  The  Small 
CHARLIE  symbols  were  8mm  wide  by  6mm  high  and  the  Large  were  1 0  by  8mm. 
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Three  different  types  of  background  were  used  in  the  experiment  -  Simple,  Complex  and  Map. 
Simple  backgrounds  were  digital  photographs  of  rural  areas.  They  had  little  variation  in  colour, 
shading,  or  geometry  and  featured  very  few  buildings  or  roads,  if  any.  Typical  images  included 
fields,  meadow,  desert  and  scrub.  Complex  backgrounds  were  provided  by  digital  photographs  of 
major  conurbations.  They  featured  extensive  colour  variation,  light  and  shading,  with  a  variety  of 
buildings  of  different  shapes  and  sizes.  Complex  backgrounds  typically  included  major  highways 
and  varied  between  suburban  and  downtown  areas  as  well  as  industrial  zones  including 
warehouses  and  plants.  All  images  used  for  the  Simple  and  Complex  backgrounds  were  presented 
at  a  scale  of  1-25000  feet.  Map  backgrounds  were  digitally  scanned  military  paper  maps  used  by 
Canadian  and  Norwegian  land  forces  and  the  US  Marine  Corps.  The  maps  were  shown  at  the 
same  scale  (1-25000  feet)  as  the  photographs. 

The  backgrounds  used  in  the  experiment  were  created  by  selecting  small  sections  from  a  few 
large  images  and  maps  and  presenting  them  either  in  their  original  orientation  or  rotated  90  or  180 
degrees.  With  this  process,  it  was  possible,  with  one  exception,  to  present  a  slightly  different 
background  on  almost  every  trial  while  maintaining  backgrounds  of  similar  complexity  across  all 
trials  in  each  background  type.  The  exception  was  that  the  same  images  were  used  in  equivalent 
HR  and  LR  trials  (e.g.  the  same  Complex  background  would  be  used  for  the  Large  CHARLIE 
symbol,  Company  Commander,  in  the  HR  and  LR  conditions).  This  was  done  because  of  the 
limited  time  available  to  generate  all  of  the  stimuli.  For  each  HR  trial,  19  distractor  symbols, 
randomly  selected  from  the  target  symbol  set,  were  randomly  place  on  a  background.  This  image, 
background  plus  symbols,  was  then  saved.  For  the  equivalent  low  resolution  condition,  each 
image  was  edited  to  mimic  the  effect  of  a  lower  resolution  screen  (50  pixels  per  inch  or  20  pixels 
per  cm  as  opposed  to  72  pixels  per  inch  or  28  pixels  per  cm)  while  keeping  the  image  size  in  the 
two  resolutions  equal.  All  images  had  a  diagonal  width  on  the  screen  of  18.2  cm.  Examples  of  the 
symbols  and  backgrounds  are  shown  below  in  Figures  2  through  4. 


Figure  2:  Example  of  the  Small  HR  ALPHA  symbols  against  a  HR,  Simple  background. 
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Figure  3:  Example  of  Large  LR  BRA  VO  symbols  displayed  against  a  LR  Complex  background. 


Figure  4:  Example  of  HR  Small  CHARLIE  symbols  against  a  HR  Map  background. 
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Task 


The  participant’s  task  on  each  trial  was  to  identify  the  target  symbol  as  quickly  as  possible  using  a 
mouse  to  position  a  cursor  over  the  symbol  they  thought  was  the  target  and  clicking  the  left 
mouse  button. 

Procedure 

The  experiment  was  conducted  over  three  sessions  with  a  different  symbol  set  (ALPHA, 
BRAVO,  or  CHARLIE)  being  evaluated  on  each  day.  The  room  was  illuminated  using  a  mixture 
of  artificial  light  (from  ceiling  lights  and  through  a  single  glazed  door).  Four  workstations  were 
set  up  with  dividers  separating  the  participants.  Viewing  distance  to  the  screen  was  approximately 
50  cm. 

At  the  start  of  the  first  session,  participants  were  given  an  orientation  briefing  on  the  overall 
study,  its  objectives,  and  what  they  woidd  be  asked  to  do.  They  were  also  given  a  sheet 
containing  all  the  relevant  target  symbols  for  the  current  session.  Following  the  initial  briefing, 
they  were  asked  to  read  an  information  sheet  about  the  study  and  to  complete  an  informed 
consent  form  The  protocol  followed  for  this  experiment  was  approved  by  the  DRDC  Ethics 
Committee  (Protocol  number:  F725).  After  participants  had  completed  the  preliminary  paper 
work  at  the  start  of  the  session,  they  were  informed  that  they  could  take  a  short  break  during  any 
natural  break  in  the  test  session  (i.e.,  when  the  next  set  of  stimuli  was  being  configured  by  the 
experimenter). 

Following  the  briefing,  participants  were  trained  to  recognise  all  of  the  symbols  within  the  set  that 
they  would  be  tested  on  during  that  session  using  a  series  of  PowerPoint  slides.  The  training 
culminated  in  an  assessment  of  the  participants’  ability  to  readily  recognise  each  of  the  seven 
symbols  within  a  particular  symbol  set.  When  participants  felt  confident  that  they  could  readily 
recognize  all  seven  symbols  within  the  set,  they  moved  on  to  the  experimental  phase  of  the  study. 
Training  was  repeated  as  and  when  necessary  to  ensure  that  the  participants  had  reached  an 
appropriate  threshold  of  knowledge.  However,  the  ability  to  readily  recognize  each  symbol  was 
not  critical  as  a  table  with  each  of  the  symbols  (with  example  modifier  information)  was 
displayed  on  the  screen  while  participants  were  carrying  out  the  task. 

The  experimental  software  package  RESOFVE,  (developed  by  Array  Systems)  provided  the 
platform  for  the  experiment.  At  the  start  of  each  trial,  the  software  presented  a  statement  such  as, 
“Find  the  Company  2i/c  (1-3),”  together  with  an  example  of  the  target  symbol  (Figure  5). 


Figure  5:  Example  of  the  screen  presented  at  the  beginning  of  each  trial. 
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When  the  participant  was  ready,  he  or  she  would  position  the  cursor  on  the  “Show  Map”  button 
and  click  the  left  mouse  button.  This  brought  up  a  display  containing  that  target.  Once  the  target 
had  been  selected,  the  test  screen  disappeared  and  the  next  target  statement  appeared. 
Participants’  responses  (hits  and  misses)  and  speed  of  response  were  recorded  by  RESOLVE  in 
an  Excel  spreadsheet. 

At  the  end  of  each  experimental  session,  participants  were  presented  with  a  series  of  examples 
(individual  or  group),  from  the  relevant  symbol  set,  which  they  were  asked  to  identify  to  assess 
the  extent  to  which  they  had  learned  the  symbols. 


Statistical  analysis 

The  results  of  interest  were  accuracy  and  response  time  per  trial.  Since  the  response  times  were 
somewhat  skewed  (skewness  was  greater  than  2  across  most  conditions),  loge  response  times 
were  used  in  all  the  analyses  reported  in  the  paper.  Often  in  visual  search  tasks,  only  the  response 
time  of  correct  trials  are  analysed.  Since  participants  were  required  to  select  a  symbol  on  each 
trial,  the  distribution  of  response  times  for  correct  and  incorrect  trials  was  similar.  Thus,  all 
response  times  were  included  in  the  analyses.  In  addition  to  the  separate  analyses  of  accuracy  and 
loge  response  time,  multivariate  analyses  of  log  response  times  and  accuracy  combined  were  also 
carried  out.  The  Pillai's  Trace  statistic  is  reported  for  the  multivariate  analysis  because  it  proved 
to  be  the  most  conservative  estimate  of  significance.  The  multivariate  analyses  allowed  us  to 
assess  any  trade-off  between  accuracy  and  response  time.  For  all  statistical  tests,  a  significance 
level  of  0.01  was  used.  The  Scheffe  test  was  used  for  post-hoc  tests  in  the  univariate  analyses. 
Separate  repeated  measures  ANOVAs  were  carried  out  on  the  detection  and  the  legibility  data. 


Results 

The  main  purpose  of  this  study  was  to  compare  the  detectability,  discriminability,  and  legibility 
of  the  three  symbol  sets.  The  remaining  variables  were  included  to  determine  how  detection  and 
discrimination  might  be  affected  by  the  display  characteristics;  performance  was  assessed  at 
different  display  resolutions,  symbol  sizes  and  backgrounds.  Thus,  the  results  will  focus  on  the 
effect  of  symbol  set  on  accuracy  and  response  times  and  the  interactions  between  symbol  set  and 
the  remaining  variables.  Performance  on  the  two  tasks  are  analysed  separately. 


Detection  performance 

As  can  be  seen  in  Table  7,  all  of  the  independent  variables,  symbol  set,  resolution,  symbol  size 
and  background,  had  a  significant  effect  on  both  accuracy  and  response  time  for  the  detection 
task.  In  addition  as  shown  by  the  results  of  the  multivariate  analysis,  all  of  the  independent 
variables  had  a  significant  effect  on  accuracy  and  response  time  combined  indicating  that  there 
was  no  trade  off  between  accuracy  and  response  time.  In  general,  performance  was  significantly 
more  accurate  and  faster  with  the  CFIARL1E  symbols  than  with  the  ALPHA  and  BRAVO 
symbols.  As  well,  performance  was  better  in  the  HR  condition  than  the  LR  condition  and  with  the 
Large  symbols.  The  overall  effect  of  background  was  relatively  small.  However,  as  Table  4  also 
shows,  there  were  several  significant  two-  three-  and  four-way  interactions.  Some  of  the  higher 
order  interactions  might  have  been  affected  by  the  fact  that  the  order  of  the  different  conditions 
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(with  the  exception  of  symbol  set)  was  not  randomized  across  participants.  Since  the  interactions 
may  mitigate  the  main  effects,  the  result  section  focuses  on  the  significant  three-way  interactions. 

Table  7:  Summary  of  repeated  measures  analyses  of  variance  for  symbol  set  including  significant 
interactions  with  resolution,  symbol  size,  and  background.  Results  for  accuracy,  response  time, 
and  a  multivariate  analysis  of  both  measures  are  shown.  Column  2  shows  the  univariate  and 

multivariate  degrees  of  freedom. 


Condition 

Univariate  / 
multivariate 
degrees  of 
freedom 

F  values  (p  <  0.01) 

Accuracy 

Response  time 

Multivariate 

Symbol  set  (SS) 

2,48  /  4,96 

40.5 

189.6 

21.6 

Resolution  (R) 

1,24/2,23 

74.4 

73.9 

74.1 

Symbol  size  (SZ) 

1,24/2,23 

21.1 

9.6 

11.0 

Background  (B) 

2,48  /  4,96 

5.6 

15.7 

7.6 

SS  *  R 

2,48  /  4,96 

39.0 

17.7 

15.7 

SS  *SZ 

2,48  /  4,  96 

n.s. 

n.s. 

n.s 

SS*B 

4,96/8,192 

7.5 

8.9 

8.4 

R*SZ 

1,24/2,23 

27.4 

n.s. 

14.7 

R*B 

2,48  /  4,96 

n.s. 

10.9 

6.0 

SZ*B 

2,48/4,96 

n.s. 

n.s. 

n.s. 

SS*R*SZ 

2,48  /  4,96 

7.2 

n.s. 

5.9 

SS*R*B 

4,96/8,192 

12.2 

5.1 

5.7 

SS*SZ*B 

4,96/8,192 

7.9 

n.s. 

4.3 

R*SZ*B 

2,48  /  4,96 

6.5 

13.9 

6.5 

SS*R*SZ*B 

4,96/8,192 

5.5 

n.s. 

3.9 

The  interaction  between  symbol  set,  resolution,  and  background  is  shown  in  Figure  6  for  both 
accuracy  and  response  time.  As  indicated,  accuracy  and  response  time  for  detection  of  the 
CHARLIE  symbols  does  not  tend  to  be  affected  by  either  resolution  or  background.  Moreover, 
response  time  is  much  faster  for  the  CHARLIE  symbols  than  for  either  the  APLHA  or  BRAVO 
symbols.  For  both  the  ALPHA  and  BRAVO  symbols,  detection  tends  to  be  more  accurate  and 
faster  in  the  HR  conditions.  However,  the  interaction  between  resolution  and  background  tends  to 
be  somewhat  different  for  the  two  symbol  sets.  With  the  ALPHA  symbols  there  is  a  moderate  but 
consistent  decrement  in  performance  across  all  three  backgrounds.  With  the  BRAVO  symbols, 
performance  is  similar  in  both  the  HR  and  LR  conditions  against  a  Simple  background,  but 
resolution  has  a  very  large  effect  on  both  accuracy  and  response  time  with  the  Map  and  Complex 
backgrounds. 

As  well,  Figure  6  illustrates  the  two-way  interactions  between  symbol  set  and  resolution,  symbol 
set  and  background,  and  resolution  and  background.  In  general,  the  two  way  interactions 
involving  symbol  set  can  be  explained  by  the  fact  that  only  the  ALPHA  and  BRAVO  symbols  are 
affected  by  either  resolution  or  background  with  resolution  having  a  much  stronger  effect  than 
background.  In  the  HR  conditions,  accuracy  is  similar  across  all  three  symbol  sets,  but  response 
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time  is  still  significantly  faster  with  the  CHARLIE  symbols.  In  terms  of  the  two  way  interaction 
between  resolution  and  background,  the  effect  of  resolution  appears  to  be  greater  with  the  Map 
and  Complex  backgrounds.  As  discussed  above,  this  is  largely  confined  to  the  BRAVO  symbol 
set. 


Symbol  set 


ALPHA  BRAVO  CHARLIE 


Symbol  set 


□  HR-Simple 

□  LR-Simple 
H  HR-Map 

□  LR-Map 

0  HR-Complex 

□  LR-Complex 


□  HR-Simple 

□  LR-Simple 
□HR-Map 
□LR-Map 
EIHR-Complex 
□LR-Complex 


Figure  6:  Accuracy  and  loge  response  time  for  the  three  symbol  sets  as  a  function  of  resolution 
and  background.  The  standard  error  bars  are  also  shown  in  this  and  subsequent  figures. 


20 


DRDC  Toronto  CR  20 1 0- 1 52 


As  with  the  previous  interactions,  any  effects  of  symbol  size  were  limited  to  ALPHA  and 
BRAVO  (Figure  7).  In  addition,  the  effects  of  symbol  size  tended  to  be  limited  to  accuracy.  With 
the  three-way  interaction  across  symbol  set,  resolution  and  symbol  size,  the  decrement  in 
accuracy  in  the  LR  condition  is  mitigated  somewhat  by  the  use  of  Large  symbols  for  ALPHA  and 
BRAVO. 


ALPHA  BRAVO  CHARLIE 


Symbol  set 


Symbol  set 


□  HR-Large 

□  HR-Small 
0  LR-Large 

□  LR-Small 


■  Large-Simple 

□  Small-Simple 

□  Large -Map 

□  Small-Map 

□  Large-Complex 

□  Small-Complex 


Figure  7:  Response  accuracy >  for  the  three  symbol  sets  as  a  function  of  a)  resolution  and  symbol 

size  and  b)  symbol  size  and  background. 


The  three-way  interaction  across  symbol  set,  background,  and  symbol  size  for  accuracy  (Figure 
7b)  is  probably  due  to  the  inconsistent  effect  of  symbol  size  across  the  three  backgrounds  for  the 
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ALPHA  and  BRAVO  symbols.  In  most  cases,  accuracy  is  slightly  better  with  the  Large  symbols 
although  the  only  clear  difference  is  with  the  BRAVO  symbols  against  a  Complex  background. 

The  significant  interaction  across  background,  resolution  and  symbol  size  is  illustrated  in  Figure 
8.  With  a  Simple  background  accuracy,  decreases  and  response  time  increases  under  the  LR 
condition  with  Small  symbols.  With  the  other  two  backgrounds,  accuracy  decreases  and  response 
time  increases  in  the  LR  condition  even  if  Large  symbols  are  used. 


1 

0.8 

I1  0-6 

{  0.4 
0.2 
0 

Simple  Map  Complex 

Background 


□  HR-Large 

□  HR-Small 
ELR-Large 

□  LR- Small 


Simple  Map  Complex 

Background 

Figure  8:  Accuracy  and  loge  response  time  across  background,  resolution  and  symbol  size. 
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□  HR-Large 

□  HR-Small 
B  LR-Large 
OLR-Small 


Although  there  was  a  significant  four-way  interaction  for  accuracy,  it  provided  little  additional 
information  about  the  interactions  among  the  different  independent  variables.  As  with  the  three 
way  interactions,  accuracy  was  consistently  high  with  the  CHARLIE  symbols.  With  the  other  two 
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symbol  sets,  accuracy  tended  to  fall  off  under  the  low  resolution,  Small  symbol  conditions 
especially  with  the  Map  and  Complex  backgrounds. 

Legibility 

As  stated  earlier,  the  primary  difference  between  the  detection  and  legibility  task  was  that 
participants  had  to  be  able  to  discriminate  the  numbers  used  to  identify  the  unit  and/or  sub-unit  of 
the  soldiers  represented  by  the  symbols  being  displayed  in  the  legibility  task.  Legibility 
performance  was  not  assessed  under  the  LR  condition  due  to  limitations  in  time  and  resources. 

As  can  be  seen  in  Table  8,  the  independent  variables  tended  to  have  less  effect  on  accuracy  with 
the  legibility  task  compared  to  the  detection  task.  This  is  not  surprising  given  that  much  of  the 
differences  in  accuracy  found  with  the  detection  task  were  related  to  lower  accuracy  in  the  low 
resolution  conditions  for  the  ALPHA  and  BRAVO  symbol  sets.  As  with  the  detection  task,  there 
was  an  overall  effect  of  symbol  set  and  background  on  response  time  and  of  symbol  size  on 
accuracy.  Again,  the  post-hoc  showed  that  response  times  were  shorter  for  the  CHARLIE 
symbols  than  the  ALPHA  and  BRAVO  symbols.  In  addition,  response  times  for  BRAVO  were 
significantly  shorter  than  for  ALPHA.  On  the  other  hand,  performance  was  significantly  more 
accurate  and  shorter  for  the  Map  background  relative  to  the  others  with  the  legibility  task  and 
performance  was  significantly  more  accurate  with  the  small  symbols.  Response  times  tended  to 
favour  the  large  symbols,  but  the  differences  were  not  significant.  All  of  the  two-way  interactions 
were  significant,  but  again  only  for  response  time.  Although  the  interaction  between  symbol  set 
and  background  for  accuracy  was  significant  overall,  a  post  hoc  showed  no  significant  differences 
among  the  nine  combinations  of  background  and  symbol  set. 

Table  8:  Summary  of  repeated  measures  analyses  of  variance  for  the  legibility  task  as  a  function 
of  symbol  set,  symbol  size,  and  background.  Results  for  accuracy,  response  time,  and  a 
multivariate  analysis  of  both  measures  are  shown.  Column  2  shows  the  univariate  and 

multivariate  degrees  of  freedom. 


Condition 

Univariate  / 
multivariate 
degrees  of 
freedom 

F  values  (p  <  0.01) 

Accuracy 

RT 

Multivariate 

Symbol  set  (SS) 

2,48  /  4,96 

n.s. 

103.03 

19.43 

Symbol  size  (SZ) 

1,24/2,23 

11.73 

n.s. 

8.31 

Background  (B) 

2,48  /  4,96 

5.13 

14.86 

6.24 

SS  *SZ 

2,48  /  4,96 

n.s. 

10.9 

4.83 

SS*B 

4,96/8,192 

3.92 

9.11 

6.29 

SZ*B 

2,48  /  4,96 

n.s. 

18.5 

6.69 

SS*SZ*B 

4,96/8,192 

n.s. 

n.s. 

n.s. 

The  interactions  between  symbol  set  and  background  and  symbol  set  and  symbol  size  are  shown 
in  Figure  9.  As  with  the  detection  task,  a  post  hoc  analysis  showed  that  response  times  were 
significantly  shorter  across  all  backgrounds  for  the  CHARLIE  symbol  set  relative  to  the  other  two 
symbol  sets.  For  the  symbol  set  /  background  interaction,  the  only  other  significant  effect  was  that 
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response  times  for  ALPHA  against  a  complex  background  were  significantly  longer  than 
response  times  for  ALPHA  and  BRAVO  with  a  Map  background  and  BRAVO  with  a  Complex 
background.  The  trend  was  the  same  for  the  symbol  set  /  symbol  size  interaction.  Response  times 
for  the  ALPHA  symbols  were  significantly  poorer  than  those  for  the  Bravo  symbols. 
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Figure  9:  Loge  response  time  as  a  function  of  symbol  set  and  background  and  symbol  set  and 

symbol  size. 


For  the  background  /  symbol  size  interaction,  the  post-hoc  supported  the  picture  shown  in  Figure 
10.  Response  times  were  significantly  faster  with  Large  symbols  only  against  a  Map  background. 
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Figure  10:  Loge  response  time  as  a  function  of  background  and  symbol  size. 


Discussion 

Symbol  set 

Based  on  the  results,  the  CHARLIE  symbol  set  was  clearly  superior  to  both  the  ALPHA  and 
BRAVO  sets.  In  addition,  detectability  and  legibility  of  the  CHARLIE  set  was  not  significantly 
affected  by  any  of  the  manipulations.  With  a  high  resolution  display,  accuracy  of  the  three  symbol 
sets  was  similar.  However,  as  resolution  decreased,  accuracy  of  ALPHA  and  BRAVO  decrease 
significantly  especially  against  the  Complex  and  Map  backgrounds  and  with  the  smaller  symbols. 
Under  all  conditions,  response  times  were  on  average  more  than  twice  as  long  with  the  ALPHA 
and  BRAVO  symbols  as  compared  to  the  CHARLIE  symbols  (3.3  seconds  on  average  for  the 
CHARLIE  symbols  as  compared  to  6.6  seconds  for  the  ALPHA  and  BRAVO  symbols). 

Thus,  it  would  seem  that  any  symbols  for  representing  the  dismounted  soldier  should  use  the 
prototype  CHARLIE  symbols  as  a  starting  point.  The  primary  difference  between  CHARLIE  and 
the  other  two  sets  was  that  all  of  the  critical  information  required  to  identify  a  particular  symbol 
was  located  within  the  symbol  border.  With  the  BRAVO  symbols,  in  particular,  information 
about  echelon  and  affiliation  appeared  against  the  image  background.  If  the  background  was  dark, 
the  contrast  between  that  part  of  the  symbol  and  the  background  was  very  low  making  symbol 
identification  difficult.  It  was  thought  that  the  use  of  the  white  flag  with  the  ALPHA  symbol  set 
would  make  this  information  more  visible.  However,  this  modification  did  not  lead  to  faster 
response  times.  In  fact,  in  the  legibility  task,  response  times  for  BRAVO  symbols  were  actually 
faster  than  those  for  ALPHA  and  in  the  detection  task,  performance  was  similar.  Thus,  the  use  of 
this  type  of  mechanism  is  not  recommended  without  further  evaluation. 
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Resolution 


Even  though  performance  with  the  CHARLIE  symbol  set  was  similar  in  both  the  HR  and  LR 
conditions,  resolution  did  impact  the  detectability  of  the  ALPHA  and  BRAVO  symbols. 
Accuracy  tended  to  be  lower  and  response  times  longer  in  the  low  resolution  conditions.  One 
possible  reason  is  that  the  echelon  information  was  encoded  using  narrow  lines.  High  spatial 
frequency  information  is  more  likely  to  be  adversely  affected  as  resolution  decreases.  For  that 
reason,  it  is  advisable  to  minimize  the  requirement  to  discriminate  fine  detail  in  designing 
symbols. 

Symbol  size 

Increasing  symbol  size  had  a  relatively  small  effect  on  performance.  Primarily  it  helped  offset  the 
decrement  in  performance  with  low  resolution  displays.  This  finding  would  be  consistent  with  the 
previous  hypothesis  that  low  resolution  displays  affect  the  discrimination  of  high  spatial 
frequency  information.  Thus,  if  it  is  necessary  to  use  a  lower  resolution  display,  symbol  size 
should  be  increased.  Unfortunately,  this  has  the  potential  to  increase  clutter.  Thus,  it  would  be 
even  more  important  to  provide  tools  for  decluttering  the  display. 


Background 

Since  future  systems  will  be  used  in  a  wide  range  of  environments,  the  symbols  must  be  visible 
over  a  wide  range  of  backgrounds.  Thus,  the  issue  in  comparing  backgrounds  was  to  determine 
the  best  means  of  minimizing  the  effect  of  background  on  detectability.  Based  on  the  results,  the 
design  guidelines  followed  in  developing  the  CHARLIE  symbol  set  achieved  that  goal.  However, 
even  with  the  other  symbol  sets,  background  did  not  have  a  large  or  consistent  affect  on 
detectability.  For  example,  with  the  ALPHA  set  detection  tended  to  be  less  accurate  with  a 
Simple  background  while  with  the  BRAVO  set,  detection  tended  to  be  poorer  with  a  Complex 
background,  as  might  be  expected.  One  possible  reason  for  this  was  the  fact  that  each  target 
symbol  was  presented  in  a  single  location  for  each  combination  of  conditions  across  all 
participants.  Thus,  some  of  the  higher  level  interactions  could  have  been  due  in  part  to  the 
location  of  the  target  symbol  under  a  specific  combination  of  conditions  instead  of  to  the 
combination  of  independent  variables  under  study.  For  example,  the  relatively  low  accuracy  for 
the  Small  ALPHA  symbols  in  the  LR  condition  against  a  Simple  background  was  due  to 
extremely  low  accuracy  for  two  symbols.  If  the  location  of  the  target  symbol  had  been 
randomized  across  participants  or  if  each  participant  had  carried  out  multiple  trials  under  each 
combination  of  conditions,  with  the  target  in  different  locations  on  each  trial,  there  might  have 
been  a  stronger  and  more  consistent  effect  of  background. 

Unfortunately,  the  time  and  resources  were  not  available  to  either  prepare  the  additional  stimuli  or 
carry  out  additional  trials.  In  future  it  might  be  preferable  to  focus  on  varying  the  location  of  the 
target  symbol  across  a  small  number  of  backgrounds  rather  than  systematically  manipulating 
background. 
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Performance  of  individual  symbols 


As  indicated,  in  the  introduction,  the  current  CHARLIE  symbol  set  is  incomplete.  Symbols  do  not 
exist  for  all  of  the  NATO  roles  and  echelons  of  dismounted  soldiers.  Thus,  a  critical  requirement 
is  to  complete  the  symbol  set.  To  support  this  process  it  is  useful  to  look  at  the  accuracy  and 
response  times  of  the  individual  symbols.  Consistent  differences  in  performance  could  help 
inform  the  development  of  the  remaining  symbols.  A  cursory  examination  of  the  effect  of  the 
different  parameters  on  accuracy  and  response  time  for  the  individual  symbols  indicated  that 
accuracy  of  the  individual  symbols  was  relatively  similar,  but  that  there  was  some  variability  in 
response  time.  Resolution  was  the  only  parameter  that  had  any  effect  on  the  variability  in 
response  time  across  the  individual  symbols  (Figure  11).  Except  for  Platoon  Commander,  the 
symbols  with  the  longest  response  times  had  two  or  three  circles.  It  could  be  that  these  symbols 
were  less  discriminable  because  they  differed  primarily  in  the  number  of  circles.  However,  if  that 
was  the  case,  one  would  have  expected  the  response  times  for  Platoon  Commander  to  be  longer  as 
well.  Given  that  the  response  times  are  averaged  over  a  wide  range  of  backgrounds  and  two 
symbols  sizes,  they  are  presumably  reasonably  robust.  However,  at  the  very  least,  it  would  seem 
prudent  to  try  to  utilize  relatively  unique  icons  for  echelon  as  much  as  possible.  For  example,  one 
would  expect  that  if  symbols  using  the  current  icon  for  squad  (Table  4)  were  added  to  the  set,  it 
would  reduce  the  visibility  of  the  Fire  Team  Feader  symbol  as  well  as  symbols  incorporating  the 
Platoon  and  Section  icons. 
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Figure  11:  Loge  response  times  for  the  individual  target  symbols  in  the  CHARLIE  symbol  set  in 
the  detection  (HR  and  LR  conditions)  and  legibility  tasks.  From  left  to  right,  the  symbols 
represent  Company  Commander,  Company  Second  in  Command,  Fire  Team  Leader,  Platoon 
Commander,  Platoon  Sergeant,  Section  Commander,  and  Section  Second  in  Command. 
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Limitations  with  the  current  experiment 


The  differences  across  individual  symbols  suggest  that  part  of  the  reason  for  the  poorer 
performance  with  the  ALPHA  and  BRAVO  symbols  could  have  been  due  to  the  number  of 
different  distractor  symbols.  In  the  detection  task,  the  symbols  for  Gunnery  Sergeant  and 
Quartermaster  Sergeant  appeared  as  distractors  with  the  ALPHA  and  BRAVO  symbol  sets,  but 
not  with  the  CHARLIE  symbol  set.  Visual  search  experiments  by  Duncan  and  Humphreys  (1989) 
found  that  search  difficulty  is  a  function  of  both  similarity  between  target  symbols  and  non  target 
symbols  and  dissimilarity  amongst  non-target  symbols.  Since  the  number  of  non-target  symbols 
was  fixed,  there  would  have  been  greater  dissimilarity  amongst  the  non-target  symbols  with  the 
ALPHA  and  BRAVO  symbol  sets  compared  to  the  CHARLIE  symbol  sets.  However,  it  is 
unlikely  that  the  use  of  eight  versus  six  different  types  of  distractor  symbols  would  account  for 
the  large  differences  in  response  time  found  in  this  study.  Moreover,  it  is  unlikely  that  the 
additional  distractor  symbols  would  lead  to  the  pattern  of  results  seen  across  the  different  factors 
studied. 

Another  limitation,  already  discussed  above,  is  the  lack  of  randomness  in  target  location  across 
participants  within  each  combination  of  conditions.  Given  this  limitation,  the  results  of  this  study 
should  be  used  primarily  to  identify  the  factors  that  are  likely  to  have  the  most  impact  on 
performance.  Future  experiments  should  focus  on  testing  this  subset  of  factors  more  rigorously. 
Of  the  three  factors  studied,  resolution  appeared  to  have  the  greatest  effect.  Moreover,  it  was  only 
resolution  that  impacted  the  performance  of  the  individual  CHARLIE  symbols.  For  that  reason,  it 
is  recommended  that  future  studies  only  manipulate  resolution.  The  default  condition  for  the  other 
factors  should  be  the  small  symbols  and  complex  backgrounds. 
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Conclusion  and  recommendations 


At  the  request  of  NATO  LCG1,  DRDC  Toronto  conducted  two  studies  to  evaluate  candidate 
symbol  sets  to  represent  individual  dismounted  soldiers.  Prototype  symbol  sets  for  this  purpose 
have  been  developed  by  several  NATO  nations.  A  heuristic  evaluation  of  these  sets  indicated  that 
none  of  them  were  entirely  consistent  with  good  human  factors  practice.  A  new  set  (CHARLIE) 
was  developed  that  conformed  more  closely  to  human  factors  guidelines.  An  empirical  evaluation 
of  two  of  the  symbol  sets,  the  proposed  symbol  sets  CHARLIE,  and  two  variants  (ALPHA  and 
BRAVO)  of  a  set  designed  by  NATO  LCG1,  was  then  conducted.  The  results  indicated  that 
detection  and  discrimination  of  the  CHARLIE  symbols  was  significantly  faster  under  all  of  the 
conditions  tested  and  significantly  more  accurate  with  low  resolution  displays.  Moreover, 
performance  with  the  CHARLIE  symbols  was  consistent  across  all  the  conditions  tested 
(background,  resolution,  and  symbol  size). 

Based  on  the  results  of  this  study,  it  is  recommended  that  the  CHARLIE  symbol  set  be  expanded 
to  incorporate  symbols  that  represent  the  full  range  of  dismounted  soldier  roles  and  echelons. 
Once  this  has  been  accomplished,  a  modified  version  of  the  current  study  should  be  carried  out. 
As  discussed  above,  it  should  not  be  necessary  to  repeat  all  of  the  conditions  used  in  this  study. 
Instead,  it  is  recommended  that  the  study  look  at  detection  and  legibility  of  the  Small  symbols 
against  relatively  complex  backgrounds. 

If  resolution  and/or  increasing  the  number  of  different  symbols  are  shown  to  have  significant 
impact  then  methods  for  improving  the  visibility  of  the  symbol  components  should  be 
investigated.  The  simplest  method  is  to  increase  symbol  size.  However,  this  has  negative  side 
effects  including  greater  symbol  overlap  and  clutter.  An  alternative  method  would  be  to  simplify 
the  symbols  by  reducing  the  amount  of  information  encoded  in  the  basic  symbol.  This  would 
result  in  a  smaller  number  of  more  discriminable  symbols.  Additional  information  could  be 
provided  on  demand  (by  selecting  the  symbol  or  a  small  subset  of  the  display)  or  by  annotating 
the  symbol  in  some  way. 

In  addition  to  investigating  the  visibility  and  legibility  of  the  larger  symbol  set,  it  will  be 
important  to  assess  its  utility.  What  information,  represented  by  these  symbols,  is  critical  to  the 
user  employing  them?  A  better  understanding  of  the  user  requirements  for  this  type  of  symbology 
could  lead  to  more  effective  designs  in  which  the  relative  visibility  of  the  different  components  of 
a  symbol  reflects  its  criticality. 
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Annex  A  National  symbol  sets  for  individual  soldiers 
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Annex  B  Symbols  used  in  Experiment 


The  following  tables  show  all  the  symbols  used  in  the  experiment.  Each  symbol  incorporates  the 
following  dimensions: 

a)  Role:  Does  the  soldier  occupy  an  officer  or  enlisted  role? 

b)  Echelon:  At  what  echelon  is  the  soldier  employed  (Company,  Platoon,  or 
Section/Squad)? 

c)  Affiliation:  What  is  the  affiliation  of  the  unit  that  employs  the  soldier? 

For  BRAVO  and  ALPHA,  role  is  indicated  by  the  number  of  diagonal  lines  inside  the  blue  circle. 
Echelon  is  denoted  by  lines  and  cross  lines  extending  above  the  circle  and  affiliation  by  the 
convention  of  unit  of  sub-unit  (e.g.  13  indicates  the  soldier  is  a  member  of  One  Platoon,  Three 
Section)  beside  the  echelon  component.  For  CHARLIE,  a  regular  frame  indicates  an  officer  while 
an  enlisted  role  is  denoted  by  the  angled  comers.  Echelon  is  indicated  by  the  icon  inside  the 
rectangle  and  affiliation  by  the  two  numbers,  separate  by  a  dash,  below  the  icon.  The  same  unit  of 
sub-unit  convention  is  used.  A  wider  range  of  echelons  (e.g.  gunnery  sergeant)  exist  for  BRAVO 
than  CHARLIE.  For  consistency,  target  symbols  were  restricted  to  the  echelons  available  in  the 
CHARLIE  set.  However,  the  additional  BRAVO  symbols  were  used  as  distractors. 

For  the  legibility  task,  at  least  two  versions  of  each  target  symbol,  representing  the  same  position 
in  different  units  (e.g.  Section  and  Platoon),  were  generated.  The  tables  also  show  the  symbols 
available  in  the  Large  and  Small  version  of  each  set. 
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Table  B9:  Available  ALPHA  symbols 
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Table  BIO:  Available  BRA  VO  symbols 
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Table  Bll:  Available  CHARLIE  symbols 


NATO  position 
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List  of  symbols/abbreviations/acronyms/initialisms 


DRDC 

Defence  Research  &  Development  Canada 

DRDKIM 

Director  Research  and  Development  Knowledge  and  Information 
Management 

LCG1 

Land  Capability  Group  1 

NATO 

North  Atlantic  Treaty  Organization 

R&D 

Research  &  Development 
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